COVID-19 Impact on Family Medicine Residents Exam Performance.

BACKGROUND AND OBJECTIVES: The COVID-19 pandemic began interrupting family medicine residency training in spring 2020. While a decline in scores on the American Board of Family Medicine In-Training Examination (ITE) has been observed, whether this decline has translated into the high-stakes Family Medicine Certification Examination (FMCE) is unclear. The goal of this study was to systematically assess the magnitude of COVID-19 impact on medical knowledge acquisition during residency, as measured by the ITE and FMCE. METHODS: A total of 19,101 initial certification candidates from 2017 to 2022 were included in this study. Annual ITE scores and FMCE scores were reported on the same scale (200-800) and served as the outcome measure. We conducted multilevel regression analysis to determine ITE score growth and FMCE scores compared to cohorts prior to COVID-19. RESULTS: During COVID-19, the increase in ITE scores from postgraduate year 2 (PGY-2) to PGY-3 was 25.5 points less, representing a 57.6% relative decrease; and from PGY-3 ITE to FMCE, it was 8.6 points less, a 12.7% relative decrease, compared with cohorts prior to COVID-19. FMCE scores were 6.6 points less during COVID-19, representing a 1.2% relative decline from the average FMCE score prior to COVID-19. CONCLUSIONS: This study found nonsubstantive COVID-19 impact on FMCE scores, but a considerable knowledge acquisition decline during residency, especially during the PGY-2 to PGY-3 period. While COVID-19 impacted learning, our findings indicated that residencies were largely able to remediate knowledge deficits before residents took the FMCE.

Differences in Physician Performance and Self-rated Confidence on High- and Low-Stakes Knowledge Assessments in Board Certification.

INTRODUCTION: Evidence links assessment to optimal learning, affirming that physicians are more likely to study, learn, and practice skills when some form of consequence ("stakes") may result from an assessment. We lack evidence, however, on how physicians' confidence in their knowledge relates to performance on assessments, and whether this varies based on the stakes of the assessment. METHODS: Our retrospective repeated-measures design compared differences in patterns of physician answer accuracy and answer confidence among physicians participating in both a high-stakes and a low-stakes longitudinal assessment of the American Board of Family Medicine. RESULTS: After 1 and 2 years, participants were more often correct but less confident in their accuracy on a higher-stakes longitudinal knowledge assessment compared with a lower-stakes assessment. There were no differences in question difficulty between the two platforms. Variation existed between platforms in time spent answering questions, use of resources to answer questions, and perceived question relevance to practice. DISCUSSION: This novel study of physician certification suggests that the accuracy of physician performance increases with higher stakes, even as self-reported confidence in their knowledge declines. It suggests that physicians may be more engaged in higher-stakes compared with lower-stakes assessments. With medical knowledge growing exponentially, these analyses provide an example of the complementary roles of higher- and lower-stakes knowledge assessment in supporting physician learning during continuing specialty board certification.

Practice Adjustments Made by Family Physicians During the COVID-19 Pandemic.

INTRODUCTION: COVID-19 pandemic-related health care disruptions necessitated rapid adaptation among family physicians to safely meet patient needs while protecting themselves and their staff. On April 1, 2020, the American Board of Family Medicine (ABFM) introduced a COVID Performance Improvement (PI) activity for physicians to report on and receive Family Medicine certification credit for practice adjustments they made during the early stages of the pandemic. We aimed to understand the types of interventions implemented, and lessons physicians learned from the efforts. METHODS: We analyzed data from COVID-PI activities submitted by self-selected family physicians between April 1 and June 30, 2020. We summarized the COVID-related topics chosen for improvement and performed a qualitative content analysis on a random sample of open-text responses about lessons learned. RESULTS: The most common practice changes among 1259 unique COVID-PI activity submissions related to virtualization of patient visits, implementing new workflows, developing screening protocols, and obtaining and preserving personal protective equipment. We identified 12 themes regarding lessons learned, most commonly regarding patient and staff safety, modified practice processes and workflows, positive perceptions of and future plans for virtual visits, access to care, and patient satisfaction. Most submitters noted early successes with their interventions. CONCLUSION: A PI activity template designed for continuous board certification allowed family physicians to report on how they successfully implemented short term practice changes during the early stages of the COVID-19 pandemic. Reflections from this subset of physicians regarding lessons learned may prove useful in informing future COVID-19 related practice changes.

Racial/Ethnic Group Trajectory Differences in Exam Performance Among US Family Medicine Residents.

BACKGROUND AND OBJECTIVES: Racial/ethnic score disparities on standardized tests are well documented. Such differences on the American Board of Family Medicine (ABFM) certification examination have not been previously reported. If such differences exist, it could be due to differences in knowledge at the beginning of residency or due to variations in the rate of knowledge acquisition during residency. Our objective was to examine the residents' mean initial scores and score trajectories using the In-Training Examination (ITE) and certification examination. METHODS: A total of 17,275 certification candidates from 2014 to 2019 were included in this study. Annual ITE scores and certification examination scores are reported on the same scale and serve as the outcome. We conducted multilevel longitudinal regression to determine initial knowledge and growth in knowledge acquisition during residency by race/ethnicity categories. RESULTS: The mean postgraduate year 1 (PGY-1) ITE score was 393.3, with minority residents scoring 16.2 to 36.0 points lower compared to White residents. The mean increase per year in exam performance from PGY-1 ITE to the certification exam was 39.9 points (95% CI, 38.7, 41.1) with additional change among race/ethnicity categories per year of -3.2 to 1.9 points. CONCLUSIONS: This study found that there were initial score disparities across race/ethnicity groups in PGY-1, and these disparities continued at the same rate throughout residency training, suggesting equality in acquisition of knowledge during family medicine residency training but with a persistent gap throughout training.

Racial/Ethnic Representation Among American Board of Family Medicine Certification Candidates from 1970 to 2020.

BACKGROUND: Because improved patient outcomes and experiences have been associated with health care workforce diversity, efforts to create a diverse family physician workforce have increased. However, a metric that could properly measure family physician representation in various contexts has seldom been studied. OBJECTIVE: The goal of this study is to propose a new metric logRQ and use it to examine the diversification progress of American Board of Family Medicine (ABFM) certification candidates relative to national, state, and historic populations, as well as medical school matriculants. METHODS: We obtained race/ethnicity for certification candidates from the 2014 to 2020 ABFM Certification Examination Registration questionnaire and examined racial/ethnic representation relative to various populations via logRQs. RESULTS: The total sample comprised 26,368 initial certification candidates and 55,347 continuing certification candidates. Asian, Hispanic, and Black's logRQ increased by 0.51, 0.42, and 0.41, respectively, in initial certification candidates compared with continuing certification candidates. In addition, logRQ standard deviation ranged from 0.19 to 0.87 across States, indicating state-level variation. Although Black and Hispanic remained underrepresented, the degree of underrepresentation had improved substantially across the past 5 decades, with logRQ increasing from -2.12 (Black) and -1.16 (Hispanic) in the 1970s to -0.46 (Black) and -0.68 (Hispanic) in the 2010s. The race/ethnicity logRQs of 2020 initial certification candidates relative to 2013 to 2014 medical school matriculants were all near 0, reflecting equitable representation. CONCLUSION: We utilized the proposed metric logRQ to quantify the advancement in representation among ABFM certification candidates in different contexts. The proposed logRQ may serve as a useful tool to monitor representation progress systematically.

The American Board of Family Medicine's 8 Years of Experience with Differential Item Functioning.

INTRODUCTION: Differential item functioning (DIF) procedures flag examination questions in which examinees from different subpopulations who are of equal ability do not have the same probability of answering it correctly. Few medical certification boards employ DIF procedures because they do not collect the needed data on the examinee's race or ethnicity. This article summarizes the American Board of Family Medicine's (ABFM) combined use of DIF procedures and an expert panel to review certification questions for bias. METHODS: ABFM certification examination data from 2013 to 2020 were analyzed using a DIF procedure to flag questions with possible ethnic or racial bias. The flagged questions were reviewed by a racially and ethnically diverse panel of content experts. If the panel judged the source of the DIF was not clinically relevant for the practice of family medicine, the question was removed from the examination. RESULTS: Out of the 3487 questions analyzed, 374 unique questions (11%) were flagged by DIF procedures as potentially biased. Of the flagged questions, the review panel felt 4 should be removed for fairness. DISCUSSION: Using DIF procedures and panel review can improve the quality of the board certification questions and demonstrate the organization's commitment to avoid racial or ethnic bias.